Recent advancements in quantization and mixed-precision approaches offers substantial opportunities to improve the speed and energy efficiency of Neural Networks (NN). Research has shown that individual parameters with varying low precision, can attain accuracies comparable to full-precision counterparts. However, modern embedded microprocessors provide very limited support for mixed-precision NNs regarding both Instruction Set Architecture (ISA) extensions and their hardware design for efficient execution of mixed-precision operations, i.e., introducing several performance bottlenecks due to numerous instructions for data packing and unpacking, arithmetic unit under-utilizations etc. In this work, we bring together, for the first time, ISA extensions tailored to mixed-precision hardware optimizations, targeting energy-efficient DNN inference on leading RISC-V CPU architectures. To this end, we introduce a hardware-software co-design framework that enables cooperative hardware design, mixed-precision quantization, ISA extensions and inference in cycle-accurate emulations. At hardware level, we firstly expand the ALU unit within our proof-of-concept micro-architecture to support configurable fine grained mixed-precision arithmetic operations. Subsequently, we implement multi-pumping to minimize execution latency, with an additional soft SIMD optimization applied for 2-bit operations. At the ISA level, three distinct MAC instructions are encoded extending the RISC-V ISA, and exposed up to the compiler level, each corresponding to a different mixed-precision operational mode. Our extensive experimental evaluation over widely used DNNs and datasets, such as CIFAR10 and ImageNet, demonstrates that our framework can achieve, on average, 15x energy reduction for less than 1% accuracy loss and outperforms the ISA-agnostic state-of-the-art RISC-V cores.

In their paper Mixed-Precision Neural Networks: A Survey, Mariam Rakka, Mohammed E. Fouda, Pramod Khargonekar and Fadi Kurdahi have reviewed recent frameworks in the literature that address mixed-precision neural network training. Here, they tell us more about mixed-precision neural networks and the main findings from their survey. Mixed-precision neural networks are neural networks with varying precision (i.e., bitwidth allocation) across layers, kernels or weights. They are now gaining momentum as the need for energy-efficient and high throughput AI hardware is growing. Binary neural networks are considered the most efficient to be deployed on hardware, however, they exhibit a non-negligible drop in the model accuracy compared to floating-point neural networks which give the best accuracy and worst energy and latency efficiency.